Cooperative obstacle avoidance is one of the key technologies of Unmanned Aerial Vehicle （UAV） system. While there exist problems of formation loss， mission failure， and increasing energy consumption during the obstacle avoidance of UAV swarm. For solving these problems， a cooperative obstacle avoidance algorithm based on improved artificial potential field and consensus protocol was proposed. First， according to the control law of multi-rotor UAVs， a control protocol to keep speed and position consistency was designed， and the artificial potential field force was scaled and transformed by normalization and high-order exponents， thereby solving the problem of oscillation failure caused by the excessive variation of the potential field force. Then， the artificial potential field force was introduced to modify the expectation formation of consensus protocol for solving the control conflict problem of the combination algorithm of artificial potential field method and consensus protocol. The proposed algorithm was simulated and compared with the formation division obstacle avoidance algorithm and dynamic window obstacle avoidance algorithm in complex obstacle environment. The results show that the proposed algorithm has the average formation loss degree reduced by 82.60% and 64.38% respectively， the average failure degree of task decreased by 98.66% and 86.01% respectively， and the total length of flight path reduced by 9.95% and 17.63% respectively. It can be seen that the proposed algorithm is suitable for the complex flight environment with multiple obstacles.